feat(otel): otel sampling support in distributed tracing - #4177

Rebuild ot fields after a local probability decision

When an inbound trace was dropped and a local target_span:any sampling rule makes a new probability decision, any valid inbound rv or th prevents regeneration here. The outbound traceparent can therefore say sampled while the preserved random value remains below the preserved threshold, or a partial member can omit the local threshold entirely, causing downstream OpenTelemetry services to calculate the wrong effective sampling probability. Preserve these fields only for INHERITED; a local probability decision needs fields reconciled with its own rate and final priority.

Useful? React with 👍 / 👎.

7d5421d

Reserve space for both owned tracestate members

With a large dd member—possible because propagated Datadog tags default to a 512-byte allowance—the first append can consume nearly the entire tracestate limit and the ot append then returns false. Its failure is ignored, and the later loop deliberately skips that ot member, so the emitted header silently loses the new sampling threshold exactly on traces carrying substantial propagated tags. The limiting pass should reserve capacity for both owned members or deliberately trim/evict lower-priority data.

Useful? React with 👍 / 👎.

6841cfb

Reapply tracestate limits when constructing span links

Allowing all 32 extracted members is safe only on the outbound-header path, which runs the limiter again after rebuilding the dd member. DDTrace\SpanLink::fromHeaders() and the restart behavior instead pass this result to ddtrace_build_span_link_from_result(), where ddtrace_format_tracestate() can prepend a dd member for an extracted Datadog origin or tags without another limiting pass. An otherwise valid 32-member or 512-byte tracecontext input can therefore produce an over-limit traceState on the span link; retain the previous reservation or normalize the rebuilt link tracestate too.

Useful? React with 👍 / 👎.

cbed965

This doesn't really need to be increased to 32, we just should handle "ot" as well and not increase commas for the "ot" case.

Fixed in 53e88b1; dd and ot are excluded from the 30 vendor-member count so both owned slots remain available.

Treat limiter-approved traces as non-probability decisions

When DD_TRACE_SAMPLE_RATE enables the trace rate limiter and the limiter approves the current candidate, limited is false, so this records a pure probability decision and emits a threshold based only on sample_rate. At throughput above DD_TRACE_RATE_LIMIT, however, admission also depends on the token-bucket limiter: for example, approved traces at sample rate 1 emit th:0 even though many equivalent traces are rejected by the limiter. Downstream OpenTelemetry metrics then treat the surviving spans as unsampled and undercount traffic; any decision for which the active limiter was consulted must omit the probability threshold (or encode the combined effective probability), not only decisions the limiter rejected.

Useful? React with 👍 / 👎.

The RFC says the limiter changes attribution only when it converts a keep into a drop.

It's odd to me to do a two-step normalization here. There's no reason why the otel parsing cannot be inlined in the loop above, producing a single normalization pass.

result.tracestate is supposed to contain the values untouched by us - i.e. other vendors, which get appended (as it should - new values should come first), ddtrace_otel_sampling_update_tracestate however manipulates the string in place every time it touches it.
Let's please not store individual data (like the otel rv and th) inside the string after parsing. Store it here on result, make it properly available on the span (for inspection / manual changing / interaction with otel sdk), and rebuild it alongside the dd= tags in ddtrace_format_tracestate.

It should also help keeping the implementation tidier, keeping scanning separate from formatting.

Also, in fact, you're storing otel_sampling_rate on the span, but never storing the parsed values there, which strikes me as odd.
Because, in fact, the otel_sampling_rate seems to always be equal to the datadog sampling rate (making it redundant as is??)

Fixed in 89ab3d4:

• it now parses and removes ot during the existing normalization pass.
• parsed OTel fields are stored separately on the trace/span and rebuilt during formatting.
• extraction now parses state, while injection handles formatting.
• parsed values are stored on the span and the redundant otel_sampling_rate was removed.

Redundant condition with the if just below?

Yes, it was redundant and has been removed in 189d449

If you'd print it directly to the smart_str, you could use smart_str_append_printf(&result, "rv:%014x", random_value) instead of reimplementing hex-printing. (and then remove trailing zeroes on the threshold)

Done in af34e57; sampling fields are now formatted directly with smart_str_append_printf